Training of Perceptron Neural Network Using Piecewise Linear Activation Function

A new Perceptron training algorithm is presented, which employs the piecewise linear activation function and the sum of squared differences error function over the entire training set. The most commonly used activation functions are continuously differentiable such as the logistic sigmoid function, the hyperbolic-tangent and the arctangent. The differentiable activation functions allow gradient-based optimization algorithms to be applied to the minimization of the error. This algorithm is based on the following approach: the activation function is approximated by its linearization near the current point, hence the error function becomes quadratic and the corresponding constraint quadratic program is solved by an active set method. The performance of the new algorithm was compared with recently reported methods. Numerical results indicate that the proposed algorithm is more efficient in terms of both, its convergence properties and the residual value of the error function.